{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Project Navigation - Report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training \n",
    "\n",
    "For each training session, we construct the **agent** with different parameters\n",
    "and we run the *Deep-Q-Network* procedure **dqn** as follows:\n",
    "\n",
    "        agent = Agent(state_size=37, action_size=4, seed=1, fc1_units=fc1_nodes, fc2_units=fc2_nodes)       \n",
    "        scores, episodes = **dqn**(n_episodes = 2000, eps_start = epsilon_start, train_numb=i)  \n",
    "    \n",
    "The  the obtained weights are saved into the file 'weights_'+str(train_numb)+'.trn'.   \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deep-Q-Network algorithm\n",
    "\n",
    "The _Deep-Q-Network_ procedure **dqn** performs the external loop (by _episodes_) till the number of episodes \n",
    "reached the maximal number of episodes _n_episodes = 2000_ or the _completion criteria_ is executed.\n",
    "For the completion criteria, we check  \n",
    "\n",
    "        np.mean(scores_window) >= 13,  \n",
    "\n",
    "where _scores_\\__window_ is the array of the type deque realizing  the shifting window of length <= 100.\n",
    "The element _scores_\\__window_[i] contains the _score_ achieved by the algorithm on the episode _i_.\n",
    "\n",
    "\n",
    "In the internal loop,  **dqn** gets the current _action_ from the **agent**.\n",
    "By this _action_ **dqn** gets _state_ and _reward_ from Unity environment _env_.\n",
    "Then, the **agent** accept params _state_, _action_, _reward_, _next_\\__state_, _done_\n",
    "to the next training step. The variable _score_ accumulates obtained rewards.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mechanisms of Agent\n",
    "\n",
    "The class **Agent** is is the well-known class implementing the following mechanisms:\n",
    "\n",
    "* Two Q-Networks (local and target) using the simple neural network.\n",
    "\n",
    "        self.qnetwork_local = QNetwork(state_size, action_size, seed, fc1_units, fc2_units).to(device)\n",
    "        self.qnetwork_target = QNetwork(state_size, action_size, seed, fc1_units, fc2_units).to(device)\n",
    "\n",
    "* Replay memory (using the class ReplayBuffer)\n",
    "\n",
    "       self.experience = namedtuple(\"Experience\", field_names=[\"state\", \"action\", \"reward\", \"next_state\", \"done\"])\n",
    "       ...\n",
    "       e = self.experience(state, action, reward, next_state, done)\n",
    "       self.memory.append(e)\n",
    "     \n",
    "* Epsilon-greedy mechanism\n",
    "\n",
    "        if random.random() > eps:\n",
    "            return np.argmax(action_values.cpu().data.numpy())\n",
    "        else:\n",
    "            return random.choice(np.arange(self.action_size))\n",
    "            \n",
    "   The epsilon become a bit smaller with each episode:\n",
    "   \n",
    "        eps = max(eps_end, eps_decay*eps), \n",
    "        \n",
    "where eps_end=0.01, eps_decay = 0.996.        \n",
    "   \n",
    "* Q-learning, i.e., using the max value for all possible actions\n",
    "* Computing the loss function by MSE loss\n",
    "\n",
    "       loss = F.mse_loss(Q_expected, Q_targets)\n",
    "     \n",
    "* Minimize the loss by gradient descend mechanism using the ADAM optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Q-Network\n",
    "\n",
    "Both Q-Networks (local and target) are implemented by the class\n",
    "**QNetwork**. This class implements the simple neural network    \n",
    "with 3 fully-connected layers and 2 rectified nonlinear layers.\n",
    "The class **QNetwork** is realized in the framework of package **PyTorch**.   \n",
    "The number of neurons of the fully-connected layers are as follows:\n",
    "\n",
    " * Layer fc1,  number of neurons: _state_\\__size_ x _fc1_\\__units_, \n",
    " * Layer fc2,  number of neurons: _fc1_\\__units_ x _fc2_\\__units_,\n",
    " * Layer fc3,  number of neurons: _fc2_\\__units_ x _action_\\__size_,\n",
    " \n",
    "where _state_\\__size_ = 37, _action_\\__size_ = 8, _fc1_\\__units_ and _fc2_\\__units_\n",
    "are the input params."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training and Testing \n",
    " \n",
    "We run 5 training sessions with different parameters _fc1_\\__units_,  _fc2_\\__units_, _eps_\\__start_,\n",
    "and we save obtained weights by the function of **PyTorch**:\n",
    "\n",
    "    torch.save(agent.qnetwork_local.state_dict(), 'weights_'+str(train_numb)+'.trn') \n",
    "     \n",
    "For input: fc1_units = 48, fc2_units = 40, we get the following training output:   \n",
    "\n",
    "train_numb:  0 eps_start:  0.992   \n",
    "Episode 569: elapsed: 0:13:55.193000, Avg.Score: 13.19,  score 23.0, How many scores >= 13: 60, eps.: 0.10  \n",
    "terminating at episode : 569 ave reward reached +13 over 100 episodes   \n",
    "\n",
    "![](layers_48x40.png)\n",
    "\n",
    "\n",
    "For input: fc1_units = 96, fc2_units = 96, the following training output is as follows:\n",
    "\n",
    "train_numb:  1 eps_start:  0.993   \n",
    "Episode 606: elapsed: 0:15:28.549000, Avg.Score: 13.02,  score 17.0, How many scores >= 13: 64, eps.: 0.09   \n",
    "terminating at episode : 606 ave reward reached +13 over 100 episodes   \n",
    "\n",
    "![](layers_96x96.png)\n",
    "\n",
    "After that we go to the testing session. For each session, we load saved weights by the function of **PyTorch** \n",
    "as follows:\n",
    "\n",
    "    file_weights = 'weights_'+str(train_n)+'.trn'\n",
    "    agent.qnetwork_local.load_state_dict(torch.load(file_weights))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    \n",
    "### Test average score\n",
    "\n",
    "The function **checkWeights** calculates the accumulated score for \n",
    "the episode generated by Unity environment _env_.\n",
    "For each testing session, we run **checkWeights** several times (=6 in this version) \n",
    "to get the average score for the given set of parameters.     \n",
    "\n",
    "\n",
    "\n",
    "Train: 0, Test: 0, Episode: 569, fc1_units: 48, fc2_units: 40, eps_start: 0.992, Score: 19.0    \n",
    "Train: 0, Test: 1, Episode: 569, fc1_units: 48, fc2_units: 40, eps_start: 0.992, Score: 17.0    \n",
    "Train: 0, Test: 2, Episode: 569, fc1_units: 48, fc2_units: 40, eps_start: 0.992, Score: 16.0    \n",
    "Train: 0, Test: 3, Episode: 569, fc1_units: 48, fc2_units: 40, eps_start: 0.992, Score: 15.0    \n",
    "Train: 0, Test: 4, Episode: 569, fc1_units: 48, fc2_units: 40, eps_start: 0.992, Score: 20.0    \n",
    "Train: 0, Test: 5, Episode: 569, fc1_units: 48, fc2_units: 40, eps_start: 0.992, Score: 18.0    \n",
    "\n",
    "       Average Score:  17.5   \n",
    "=========================================================   \n",
    "\n",
    "Train: 1, Test: 0, Episode: 606, fc1_units: 96, fc2_units: 96, eps_start: 0.993, Score: 14.0       \n",
    "Train: 1, Test: 1, Episode: 606, fc1_units: 96, fc2_units: 96, eps_start: 0.993, Score: 14.0    \n",
    "Train: 1, Test: 2, Episode: 606, fc1_units: 96, fc2_units: 96, eps_start: 0.993, Score: 11.0   \n",
    "Train: 1, Test: 3, Episode: 606, fc1_units: 96, fc2_units: 96, eps_start: 0.993, Score: 17.0   \n",
    "Train: 1, Test: 4, Episode: 606, fc1_units: 96, fc2_units: 96, eps_start: 0.993, Score: 11.0   \n",
    "Train: 1, Test: 5, Episode: 606, fc1_units: 96, fc2_units: 96, eps_start: 0.993, Score: 9.0   \n",
    "\n",
    "       Average Score:  12.67   \n",
    "=========================================================    \n",
    "Train: 2, Test: 0, Episode: 577, fc1_units: 96, fc2_units: 80, eps_start: 0.992, Score: 19.0    \n",
    "Train: 2, Test: 1, Episode: 577, fc1_units: 96, fc2_units: 80, eps_start: 0.992, Score: 13.0   \n",
    "Train: 2, Test: 2, Episode: 577, fc1_units: 96, fc2_units: 80, eps_start: 0.992, Score: 14.0   \n",
    "Train: 2, Test: 3, Episode: 577, fc1_units: 96, fc2_units: 80, eps_start: 0.992, Score: 19.0   \n",
    "Train: 2, Test: 4, Episode: 577, fc1_units: 96, fc2_units: 80, eps_start: 0.992, Score: 18.0   \n",
    "Train: 2, Test: 5, Episode: 577, fc1_units: 96, fc2_units: 80, eps_start: 0.992, Score: 13.0   \n",
    "\n",
    "       Average Score:  16.0   \n",
    "=========================================================   \n",
    "\n",
    "Train: 3, Test: 0, Episode: 593, fc1_units: 80, fc2_units: 72, eps_start: 0.993, Score: 17.0   \n",
    "Train: 3, Test: 1, Episode: 593, fc1_units: 80, fc2_units: 72, eps_start: 0.993, Score: 15.0   \n",
    "Train: 3, Test: 2, Episode: 593, fc1_units: 80, fc2_units: 72, eps_start: 0.993, Score: 11.0   \n",
    "Train: 3, Test: 3, Episode: 593, fc1_units: 80, fc2_units: 72, eps_start: 0.993, Score: 11.0   \n",
    "Train: 3, Test: 4, Episode: 593, fc1_units: 80, fc2_units: 72, eps_start: 0.993, Score: 16.0   \n",
    "Train: 3, Test: 5, Episode: 593, fc1_units: 80, fc2_units: 72, eps_start: 0.993, Score: 17.0  \n",
    "\n",
    "        Average Score:  14.5   \n",
    "=========================================================   \n",
    "Train: 4, Test: 0, Episode: 586, fc1_units: 112, fc2_units: 120, eps_start: 0.993, Score: 15.0  \n",
    "Train: 4, Test: 1, Episode: 586, fc1_units: 112, fc2_units: 120, eps_start: 0.993, Score: 14.0   \n",
    "Train: 4, Test: 2, Episode: 586, fc1_units: 112, fc2_units: 120, eps_start: 0.993, Score: 21.0   \n",
    "Train: 4, Test: 3, Episode: 586, fc1_units: 112, fc2_units: 120, eps_start: 0.993, Score: 15.0   \n",
    "Train: 4, Test: 4, Episode: 586, fc1_units: 112, fc2_units: 120, eps_start: 0.993, Score: 14.0   \n",
    "Train: 4, Test: 5, Episode: 586, fc1_units: 112, fc2_units: 120, eps_start: 0.993, Score: 16.0  \n",
    "        \n",
    "        Average Score:  15.83 \n",
    "=========================================================   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solved environment (from my local machine)\n",
    "\n",
    "We get the average score > 13 in 5 from 6 testing sessions.  The number of episodes needed to reach this score \n",
    "less than 600. Only in the case fc1_units = 96, fc2_units = 96 we get the average score < 13, however in this case\n",
    "in 3 tests from 6 we get the average score > 13.  From the examples that I have seen so far (on my local machine and on\n",
    "the workspace machine), we get better results when fc1_inits > fc2_units. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Future ideas \n",
    "\n",
    "The future ideas for improving the agent's performance.\n",
    "\n",
    "1. Possible improve can be achieved by adding one or more nonlinear (or also linear) layers to the neural network.\n",
    "\n",
    "2. Certainly it can be checked more cases of different numbers of neurons per NN layer.\n",
    "   It is interesting to catch a good relation between fc1_units and fc2_inits.\n",
    "\n",
    "3. The starting value of epsilon should be parameter for improve.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References to possible improvements\n",
    "\n",
    "It would be very useful to check improvements in [Deep Q Learning: Dueling Double DQN, Prioritized Experience Replay, and fixed Q-targets](https://medium.freecodecamp.org/improvements-in-deep-q-learning-dueling-double-dqn-prioritized-experience-replay-and-fixed-58b130cc5682)\n",
    "\n",
    "One effective way to improve the performance is by using Prioritized Experience Replay. It will be interesting to check [github repo](https://github.com/rlcode/per) for a fast implementation of Prioritized Experience Replay using a special data structure Sum Tree.\n",
    "\n",
    "### The recent achievement \n",
    "\n",
    "[Open AI group to play Dota 2](https://openai.com/blog/dota-2/) using Reinforcement Learning. They have created a bot which beats the world’s top professionals at 1v1 matches of [Dota 2](http://blog.dota2.com/?l=english) under standard tournament rules. The bot learned the game from scratch by self-play, and does not use imitation learning or tree search. This is a step towards building AI systems which accomplish well-defined goals in messy, complicated situations involving real humans. \n",
    "\n",
    "### Competitive Self-Play\n",
    "\n",
    "[Push opponent outside the ring and other](https://www.youtube.com/watch?v=OBcjhp4KSgQ)\n",
    "\n",
    "### Blog\n",
    "\n",
    "[Tackling, ducking, faking, kicking, catching, and diving for the ball](https://openai.com/blog/competitive-self-play/)\n",
    "\n",
    "### Tensorflow or PyTorch\n",
    "Tensorflow is based on Theano and has been developed by Google, whereas PyTorch is based on Torch and has been developed by Facebook. [The force is strong with which one?](https://medium.com/@UdacityINDIA/tensorflow-or-pytorch-the-force-is-strong-with-which-one-68226bb7dab4)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:Anaconda_3]",
   "language": "python",
   "name": "conda-env-Anaconda_3-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
